Initial Python Repo Manager #97

clavedeluna · 2023-10-25T11:39:30Z

Overview

The purpose here is to kickstart the code for analyzing a project directory, its metadata and dependencies.

Description

a PythonRepoManager's goal is to take the parent directory passed to codemodder and to analyze that repo's files for existence of any pkg management system or dependency store, such as requirements.txt, project.toml, setup.py, setup.cfg, poetry's more special project.toml, Pipfile, conda's .yaml files.
flexibility is key here as each file type is unique and a repo may contain none or many of these.
I've added this architecture but codemodder is not running it at this time. I've also copied some code from DependencyManager which will be deduped later on. This will come soon.

There is some key work that needs to happen going forward:

Dudupe / connect PythonRepoManager with DependencyManager. I think ideally DM will be inside PRM, but we shall see as a next step. There's quite a bit of changes going on to DM right now so I'd like to sandbox it
Handle edge cases. I've added quite a few todos to each parser to handle all sorts of edge cases in each file.
Add remaining file parsers: I added a few but we still need: poetry's more special project.toml, Pipfile, conda's .yaml files.
Higher-level testing: maybe have an integration test that runs PRM on known OSS packages and see the results we get. This would add variety to see if we need edge-case handling.
Heuristics for how to handle a parent_directory with 0 or more of these file types. Where will we add dependencies we need to add? How do we handle dependencies declared in multiple files? How do we handle conflicting information? Do we only take the first found file? Lots of discussion we need here.

codecov · 2023-10-25T11:46:15Z

Codecov Report

Merging #97 (1ee9b1b) into main (e09942c) will increase coverage by 0.28%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #97      +/-   ##
==========================================
+ Coverage   95.22%   95.51%   +0.28%     
==========================================
  Files          51       60       +9     
  Lines        2285     2432     +147     
==========================================
+ Hits         2176     2323     +147     
  Misses        109      109

Files	Coverage Δ
src/codemodder/codemodder.py	`98.05% <100.00%> (+0.03%)`	⬆️
src/codemodder/context.py	`97.29% <100.00%> (+0.11%)`	⬆️
src/codemodder/dependency_manager.py	`92.45% <ø> (ø)`
...demodder/project_analysis/file_parsers/__init__.py	`100.00% <100.00%> (ø)`
...odder/project_analysis/file_parsers/base_parser.py	`100.00% <100.00%> (ø)`
...der/project_analysis/file_parsers/package_store.py	`100.00% <100.00%> (ø)`
...nalysis/file_parsers/pyproject_toml_file_parser.py	`100.00% <100.00%> (ø)`
...lysis/file_parsers/requirements_txt_file_parser.py	`100.00% <100.00%> (ø)`
...ect_analysis/file_parsers/setup_cfg_file_parser.py	`100.00% <100.00%> (ø)`
...ject_analysis/file_parsers/setup_py_file_parser.py	`100.00% <100.00%> (ø)`
... and 2 more

clavedeluna · 2023-10-25T13:58:33Z

the codecov failure isn't legitimate to me. It mentions lack of coverage on a file I didn't even touch.

andrecsilva

Minor changes. See comments.

andrecsilva · 2023-10-25T13:53:21Z

src/codemodder/project_analysis/file_parsers/setup_py_file_parser.py

+        )
+
+
+class SetupCallVisotor(cst.CSTVisitor):


Visotor -> Visitor

andrecsilva · 2023-10-25T13:59:21Z

src/codemodder/project_analysis/file_parsers/utils.py

+def clean_simplestring(node: cst.SimpleString | str) -> str:
+    if isinstance(node, str):
+        return node.strip('"')
+    return node.value.strip('"')


Use node.raw_value. It will strip the quotes and prefixes.

drdavella · 2023-10-25T14:35:39Z

poetry's more special project.toml, Pipfile, conda's .yaml files

These are good call-outs. I hope we can defer these until later and just try to handle the more "standard" cases for now.

Heuristics for how to handle a parent_directory with 0 or more of these file types. Where will we add dependencies we need to add? How do we handle dependencies declared in multiple files? How do we handle conflicting information? Do we only take the first found file? Lots of discussion we need here.

A lot of really good points here too. I think the simplest heuristic will be to have some kind of priority order that we define in advance, and make updates only to the first of those files that we detect. We can build out more functionality from that point. It feels like we're paving a new path here, though, and we're on our own to figure it out.

drdavella

This looks great. I'm sure there is going to be a lot of iteration on this and the dependency manager, but this gives us a really good foundation to build upon. Just a few minor comments but otherwise 👍

drdavella · 2023-10-25T14:37:08Z

src/codemodder/context.py

        self.directory = directory
        self.dry_run = dry_run
        self.verbose = verbose
        self._results_by_codemod = {}
        self._failures_by_codemod = {}
        self.dependencies = {}
        self.registry = registry
+        self.repo_manager = repo_manager


Very minor thing but we should add the type declaration at the class definition above.

Are you referring to something else that's not already line 43, repo_manager: PythonRepoManager,?

Yes, it should be added on line 35 above too.

drdavella · 2023-10-25T14:38:14Z

src/codemodder/project_analysis/file_parsers/base_parser.py

+    @property
+    @abstractmethod
+    def file_name(self):
+        ...  # pragma: no cover


Just a tiny thing but I think you can have the ... on the same line as the def which probably makes the # pragma unnecessary.

oh I hope so!

black doesn't like inline ... so I left it as is

src/codemodder/project_analysis/file_parsers/setup_cfg_file_parser.py

Minor requests completed

clavedeluna force-pushed the repo-manager branch from 1cb5118 to 78dbd68 Compare October 25, 2023 12:01

clavedeluna marked this pull request as ready for review October 25, 2023 12:23

clavedeluna requested review from drdavella and andrecsilva as code owners October 25, 2023 12:23

andrecsilva previously requested changes Oct 25, 2023

View reviewed changes

drdavella approved these changes Oct 25, 2023

View reviewed changes

clavedeluna added 8 commits October 25, 2023 15:14

add repo manager w reqs txt file parser

31d0ac2

add pyproject.toml parser

8bc6b5c

add parser for setup.cfg file

8de212b

add setup.py parser

9d68e03

testing

b1b8dbf

use abstract

58f0e1f

ignore cov in base class

e141197

review feedback

1ee9b1b

clavedeluna force-pushed the repo-manager branch from 66f3c48 to 1ee9b1b Compare October 25, 2023 18:14

clavedeluna requested a review from andrecsilva October 25, 2023 18:15

clavedeluna merged commit 4663b59 into main Oct 25, 2023
11 checks passed

clavedeluna deleted the repo-manager branch October 25, 2023 18:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial Python Repo Manager #97

Initial Python Repo Manager #97

clavedeluna commented Oct 25, 2023 •

edited

Loading

codecov bot commented Oct 25, 2023 •

edited

Loading

clavedeluna commented Oct 25, 2023

andrecsilva left a comment

andrecsilva Oct 25, 2023

andrecsilva Oct 25, 2023

drdavella commented Oct 25, 2023 •

edited

Loading

drdavella left a comment

drdavella Oct 25, 2023

clavedeluna Oct 25, 2023

drdavella Oct 25, 2023

drdavella Oct 25, 2023

clavedeluna Oct 25, 2023

clavedeluna Oct 25, 2023

Initial Python Repo Manager #97

Initial Python Repo Manager #97

Conversation

clavedeluna commented Oct 25, 2023 • edited Loading

Overview

Description

codecov bot commented Oct 25, 2023 • edited Loading

Codecov Report

clavedeluna commented Oct 25, 2023

andrecsilva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drdavella commented Oct 25, 2023 • edited Loading

drdavella left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clavedeluna commented Oct 25, 2023 •

edited

Loading

codecov bot commented Oct 25, 2023 •

edited

Loading

drdavella commented Oct 25, 2023 •

edited

Loading